perf: precompute org page contributors leaderboard - #4501
Conversation
Signed-off-by: anilb <epipav@gmail.com>
PR SummaryMedium Risk Overview Adds Reviewed by Cursor Bugbot for commit 56d7404. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
|
There was a problem hiding this comment.
Pull request overview
Precomputes organization contributor leaderboards nightly to reduce expensive request-time scans.
Changes:
- Adds a precomputed contributor-count datasource and scheduled copy pipe.
- Uses precomputed data for unfiltered requests while retaining date-filtered scans.
- Note: the PR title lacks the required
(CM-XXX)suffix.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
org_page_contributors.pipe |
Routes unfiltered queries to precomputed data. |
org_page_contributors_copy_pipe.pipe |
Builds the nightly contributor snapshot. |
org_page_contributors_copy_ds.datasource |
Stores per-organization contributor counts. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| {% if use_precomputed == 1 %} | ||
| SELECT count() | ||
| FROM org_page_contributors_copy_ds | ||
| WHERE organizationId = (SELECT id FROM org_slug_lookup) |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Reviewed by Cursor Bugbot for commit 56d7404. Configure here.
| SELECT organizationId, memberId, count() AS contributionCount, now() AS computedAt | ||
| FROM activityRelations_deduplicated_cleaned_bucket_union | ||
| WHERE organizationId != '' | ||
| GROUP BY organizationId, memberId |
There was a problem hiding this comment.
Contributors include stars and forks
High Severity
The new COPY aggregates every activity row per (organizationId, memberId) with no activityTypes / code-contribution filter. Stargazers, forkers, and similar engagement actors become “contributors,” which can inflate leaderboard counts by an order of magnitude. Sibling org-page COPY pipes (org_page_kpis, org_page_contributors_timeseries, org_page_projects) already restrict to code contributions.
Triggered by learned rule: Tinybird pipes counting contributors must filter by activityTypes
Reviewed by Cursor Bugbot for commit 56d7404. Configure here.
| SELECT organizationId, memberId, count() AS contributionCount, now() AS computedAt | ||
| FROM activityRelations_deduplicated_cleaned_bucket_union | ||
| WHERE organizationId != '' | ||
| GROUP BY organizationId, memberId |
There was a problem hiding this comment.
Empty memberId counted as contributor
Medium Severity
The COPY filters organizationId != '' but not memberId != '', so unresolved/empty member IDs become one row per org. The new precomputed count() paths then count that row as a real contributor, while the leaderboard INNER JOIN to members_sorted drops it—so count mode and the listed leaderboard disagree by one.
Additional Locations (2)
Triggered by learned rule: Tinybird pipes with count(DISTINCT memberId/organizationId) must filter empty string values
Reviewed by Cursor Bugbot for commit 56d7404. Configure here.
Optimizes
org_page_contributorsendpoint pipe with a nightly dedicated precompute copy pipe